Many ranking-and-selection (R&S) problems arise in settings where information is noisy, structured, and expensive. In multi-fidelity experimentation, one can query cheap but biased proxies or expensive high-fidelity measurements; in dueling bandits, feedback arrives only through pairwise comparisons rather than direct rewards. These models are increasingly natural in engineering design, simulation optimization, preference learning for LLMs, and human-in-the-loop evaluation, where absolute scores are often unavailable or prohibitively costly and decisions must be made with a prescribed level of confidence. What makes these settings especially challenging is that the usual single-winner template is no longer sufficient. First, the answer map may be set-valued: in good-alternative or subset-selection problems, several answers can be simultaneously correct. Second, even when the true instance is answerable, a noisy estimate may temporarily fall outside the answerable set.

Bayesian Deep Ensembles (BDEs) represent a powerful approach for uncertainty quantification in deep learning, combining the robustness of Deep Ensembles (DEs) with flexible multi-chain MCMC. While DEs are affordable in most deep learning settings, (long) sampling of Bayesian neural networks can be prohibitively costly. Yet, adding sampling after optimizing the DEs has been shown to yield significant improvements. This leaves a critical practical question: How long should the sequential sampling process continue to yield significant improvements over the initial optimized DE baseline? To tackle this question, we propose a stopping rule based on E-values. We formulate the ensemble construction as a sequential anytime-valid hypothesis test, providing a principled way to decide whether or not to reject the null hypothesis that MCMC offers no improvement over a strong baseline, to early stop the sampling. Empirically, we study this approach for diverse settings. Our results demonstrate the efficacy of our approach and reveal that only a fraction of the full-chain budget is often required.

In this paper we revisit the fixed-confidence identification of the Pareto optimal set in a multi-objective multi-armed bandit model.

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Neural Information Processing Systems http://nips.cc/

Learning the minimum/maximum mean among a finite set of distributions is a fundamental sub-task in planning, game tree search and reinforcement learning.

In this paper, we show that early stopping can also be applied to obtain the minimax optimal testing in a general non-parametric setup.

In this paper we revisit the fixed-confidence identification of the Pareto optimal set in a multi-objective multi-armed bandit model.

Through a new notion called "optimism" that captures certain natural monotonic